tag repronim based on https://git-annex.branchable.com/forum/Is_there_a_way_to_have_assistant_add_files_locked__63__/#comment-096bedb2d22d5aae6a51a53179372d4f
improve error message when SETCREDS overwrites git-annex config
That is not allowed, so it's not a bug in git-annex when it happens and
instead tell the special remote developer how it's messed up.
Note that currently only Remote.External can overwrite the parsed remote
config with a PassedThrough value. PassedThrough values are otherwise
only generated for configs that are not parsed by the remote config
parser.
work around file-io not setting locale encoding when opening a Handle
Works around this bug https://github.com/haskell/file-io/issues/45
The fix is in Utility.FileIO.CloseOnExec because all use of file-io is
already wrapped through that module. Although perhaps that ought to be
refactored at this point.
I'd hope that file-io will eventually fix this bug, and also provide
CloseOnExec variants of its functions. That would allow depending on the
fixed version, and removing this ugly code.
Note that, functions like readFile that don't care about the encoding
due to reading/writing a ByteString were kept optimally fast by not
setting the encoding. This avoids an IORef read and write per open.
Improve performance when used with a local git remote that has a large working tree
git write-tree was being run once per file git-annex acts on when eg,
getting files, which is slow when the remote repository has a large
tree.
onLocal calls quiesce after each action, and quiesce closes the keys db
since [[!commit ba7ecbc6a9c]]. Which has a relevant comment about
performance. I have not addressed that, the keys db still gets closed and
reopened after each file.
Turns out that, since git write-tree was run by each call to
reconcileStaged, the .git/annex/keysdb.cache value was never the
same as the git index's inode. Because git write-tree updates the index's
mtime even when no changes have been made.
And so, when the database got closed and reopened, reconcileStaged would
see a changed index, and run git write-tree again. Over and over.
I considered writing the index's new inodecache after write-tree to the
keysdb.cache, but that would be vulnerable to a race, if the index was
changed just after write-tree.
The fix was to stop using keysb.cache at all. When the database is closed
and later reopened by the same process, avoid re-doing reconcileStaged.
Now that .git/annex/keysdb.cache is no longer used. It could be removed,
but the time overhead of removing it would be more than the space overhead
of keeping it. Defferred removal to the v11 upgrade.
convert withFile and withBinaryFile to close-on-exec safe versions
At this point, every call in git-annex to withFile, openBinaryFile,
withBinaryFile, appendFile, and openTempFile, readFile, and writeFile
have been converted.
Sponsored-by: the NIH-funded NICEMAN (ReproNim TR&D3) project
Exported by Common, so they will be available everywhere.
These are the same as readFile, writeFile, appendFile.
But have two benefits:
* They take OsPath, so using them avoids converting back and forth
unncessarily.
* They use the close-on-exec flag so can't leak FDs to child processes.
Unlike the standard Haskell versions which unfortunately have that
wart currently. (I do hope the standard versions get fixed
eventually.)
Sponsored-by: the NIH-funded NICEMAN (ReproNim TR&D3) project
Utility.FileIO: set close-on-exec flag for all functions
Utility.FileIO.CloseOnExec is largely copied from
System.File.OsPath.Internal with the simple modification of
setting the flag.
Unfortunately, openTempFile does not set the flag when
opening the file, but afterwards, leaving it vulnerable to a race.
A lot of code, including posix and windows specific code,
would need to be copied from file-io in order to fix that.
Still, I consider this implementation a placeholder, it doesn't truely
fix all instances of the problem.
I hope that this will be addressed in file-io itself, see
https://github.com/haskell/file-io/issues/44
Utility.FileIO.CloseOnExec could form the basis of a
file-io-closeonexec library, depending on how things go with that
issue.
Sponsored-by: the NIH-funded NICEMAN (ReproNim TR&D3) project
It was already default in stack builds, now it is default in cabal
builds as well.
Add build warnings when git-annex is built without the OsPath build flag.
git-annex version: Report on whether it was built with the OsPath build flag.
Having the flag on by default was always the plan, and this is a good time to
make the change. A bit of added urgency comes from the close-on-exec leak
issue. Fixing that is going to need reimplentation of things like openFile.
Needing to reimplenment it twice is not very appealing, especially since the
FilePath version of it has an implementation that cannot be easily copied and
tweaked. If OsPath is on by default, I can start with only implementing
openFile for it, and fix the bug in that build. And perhaps avoid doing the
extra work that will later get thrown away when this transition finishes.
Note that at this point, Debian still needs to package file-io. Hopefully, they
will package it, rather than turning off the OsPath build flag.
convert all bytestring readFile/writeFile to use Utility.FileIO
This is groundwork for setting the close-on-exec flag when opening
files, which will be done in Utility.FileIO or a similar module using the same
function names. The bytestring library does not set that flag, even though it
could, and IMHO should.
Note that there are many calls to the Prelude's readFile/writeFile/appendFile
still in git-annex, and this does not address those.
Sponsored-by: the NIH-funded NICEMAN (ReproNim TR&D3) project
Made all uses of openFd and dup set the close-on-exec flag, with a few
exceptions when starting a git-annex daemon.
Made openFdWithMode be used everywhere, rather than openFd.
Adding a new parameter to it ensures I checked everything.
And will help to make sure this gets considered in the future when
opening fds.
In lockPidFile, the only thing that keeps the pid file locked, once
daemonize re-runs the command in a new session, is that the fd is
inherited.
In Utility.LogFile.redir, the new fd it dups to does not have the
close-on-exec flag set, because this is used to set up the stdout and
stderr fds, which need to be inherited by child processes.
Same in Assistant.startDaemon where the browser gets started with the
original stdout and stderr.
This does nothing about uses of openFile and similar!
avoid overhead of fcntl when setting close-on-exec
unix-2.8.0 adds cloexec to OpenFileFlags, rather than needing to call
setFdOption after opening.
This also might avoid a race, if another thread started a process at
just the wrong time, before the flag got set, it could inherit the FD.
Unfortunately, the arm64-ancient build needs support for older versions
of unix than that, so had to ifdef. That build is still needed to
support using git-annex in termux on phones like mine, I have confirmed
today.
add: Fix crash adding filenames that are exactly 21 bytes long and begin
with a utf-8 character.
Also longer filenames that start with "....." would cause the same crash.
I also audited for other calls to truncateFilePath that could truncate it
to "". Most use pathmax so are not a problem. Backend.Utilities.genKeyName
could possibly truncate it like that, but appends the md5 so would not be a
problem either.
projects / git-annex.git / log